Development and Evaluation of a Tutorial to Improve Students' Understanding of a Lock-in amplifier

A lock-in amplifier is a versatile instrument frequently used in physics research. However, many students struggle with the basic operating principles of a lock-in amplifier which can lead to a variety of difficulties. To improve students' understanding, we have been developing and evaluating a research-based tutorial which makes use of a computer simulation of a lock-in amplifier. The tutorial is based on a field-tested approach in which students realize their difficulties after predicting the outcome of simulated experiments involving a lock-in amplifier and check their predictions using the simulated lock-in amplifier. Then, the tutorial provides guidance and strives to help students develop a coherent understanding of the basics of a lock-in amplifier. The tutorial development involved interviews with physics faculty members and graduate students and iteration of many versions of the tutorial with professors and graduate students. The student difficulties with lock-in amplifiers and the development and assessment of the research-based tutorial to help students develop a functional understanding of this device are discussed.Comment: Currently under review for Phys Rev ST PER. arXiv admin note: text overlap with arXiv:1601.0128

USING THE TUTORIAL APPROACH TO IMPROVE PHYSICS LEARNING FROM INTRODUCTORY TO GRADUATE LEVEL

In this thesis, I discuss the development and evaluation of tutorials ranging from introductory to graduate level. Tutorials were developed based upon research on student difficulties in learning relevant concepts and findings of cognitive research. Tutorials are a valuable resource when used either in-class or as a self-study tool. They strive to help students develop a robust knowledge structure of relevant topics and improve their problem solving skills. I discuss the development of a tutorial on the Lock-in amplifier (LIA) for use as both an on-ramp to ease the transition of students entering into the research lab and to improve student understanding of the operation of the LIA for those already making use of this device. The effectiveness of this tutorial was evaluated using think aloud interviews with graduate students possessing a wide range of experience with the LIA and the findings were uniformly positive. I also describe the development and evaluation of a Quantum Interactive Learning Tutorial (QuILT) that focuses on quantum key distribution using two protocols for secure key distribution. One protocol used in the first part of the QuILT is administered to students working collaboratively in class while the second protocol used in the second part of the QuILT was administered as homework. Evaluation of student understanding of the two protocols used in this QuILT shows that it was effective at improving student understanding both immediately after working on the QuILT and two months later. Finally, I discuss the development and evaluation of four web-based tutorials focusing on quantitative problem solving intended to aid introductory students in the learning of effective problem-solving heuristics while helping them learn physics concepts. Findings suggest that while these tutorials are effective when administered in one-on-one think-aloud interviews, this effectiveness is greatly diminished when students are asked to use the tutorials as a self-study tool with no supervision. In addition, the development and evaluation of four sets of scaffolded prequizzes for introductory physics on the same topics as the tutorials is discussed. These prequizzes are designed to mimic the structure of the web-based tutorials and can be implemented in the classroom

Examining the effects of testwiseness in conceptual physics evaluations

Testwiseness is defined as the set of cognitive strategies used by a student that is intended to improve his or her score on a test regardless of the test’s subject matter. Questions with elements that may be affected by testwiseness are common in physics assessments, even in those which have been extensively validated and widely used as evaluation tools in physics education research. The potential effect of several elements of testwiseness were analyzed for questions in the Force Concept Inventory (FCI) and Conceptual Survey on Electricity and Magnetism that contain distractors that are predicted to be influenced by testwiseness. This analysis was performed using data sets collected between fall 2001 and spring 2014 at one midwestern U.S. university (including over 9500 students) and between Spring 2011 and Spring 2015 at a second eastern U.S. university (including over 2500 students). Student avoidance of “none of the above” or “zero” distractors was statistically significant. The effect of the position of a distractor on its likelihood to be selected was also significant. The effects of several potential positive and negative testwiseness effects on student scores were also examined by developing two modified versions of the FCI designed to include additional elements related to testwiseness; testwiseness produced little effect post-instruction in student performance on the modified instruments

Behavioral Self-Regulation in a Physics Class

This study examined the regulation of out-of-class time invested in the academic activities associated with a physics class for 20 consecutive semesters. The academic activities of 1676 students were included in the study. Students reported investing a semester average of 6.5 2.9 h out of class per week. During weeks not containing an examination, a total of 4.3 2.1 h was reported which was divided between 2.5. 1.2 h working homework and 1.8 1.4 h reading. Students reported spending 7.6. 4.8 h preparing for each in-semester examination. Students showed a significant correlation between the change in time invested in examination preparation (r ¼ −0.12, p \u3c 0.0001) and their score on the previous examination. The correlation increased as the data were averaged over semester (r ¼ −0.70, p ¼ 0.0006) and academic year (r ¼ −0.82, p ¼ 0.0039). While significant, the overall correlation indicates a small effect size and implies that an increase of 1 standard deviation of test score (18%) was related to a decrease of 0.12 standard deviations or 0.9 h of study time. Students also modified their time invested in reading as the length of the textbook changed; however, this modification was not proportional to the size of the change in textbook length. Very little regulation of the time invested in homework was detected either in response to test grades or in response to changes in the length of homework assignments. Patterns of regulation were different for higher performing students than for lower performing students with students receiving a course grade of “C” or “D” demonstrating little change in examination preparation time in response to lower examination grades. This study suggests that homework preparation time is a fixed variable while examination preparation time and reading time are weakly mutable variables

Using Machine Learning to Predict Physics Course Outcomes

The use of machine learning and data mining techniques across many disciplines has exploded in recent years with the field of educational data mining growing significantly in the past 15 years. In this study, random forest and logistic regression models were used to construct early warning models of student success in introductory calculus-based mechanics (Physics 1) and electricity and magnetism (Physics 2) courses at a large eastern land-grant university. By combining in-class variables such as homework grades with institutional variables such as cumulative GPA, we can predict if a student will receive less than a “B” in the course with 73% accuracy in Physics 1 and 81% accuracy in Physics 2 with only data available in the first week of class using logistic regression models. The institutional variables were critical for high accuracy in the first four weeks of the semester. In-class variables became more important only after the first in-semester examination was administered. The student’s cumulative college GPA was consistently the most important institutional variable. Homework grade became the most important in-class variable after the first week and consistently increased in importance as the semester progressed; homework grade became more important than cumulative GPA after the first in-semester examination. Demographic variables including gender, race or ethnicity, and first generation status were not important variables for predicting course grade

Multi-Dimensional Item Response Theory and the Force Concept Inventory

Research on the test structure of the Force Concept Inventory (FCI) has largely been performed with exploratory methods such as factor analysis and cluster analysis. Multi-Dimensional Item Response Theory (MIRT) provides an alternative to traditional Exploratory Factor Analysis which allows statistical testing to identify the optimal number of factors. Application of MIRT to a sample of $N=4,716$ FCI post-tests identified a 9-factor solution as optimal. Additional analysis showed that a substantial part of the identified factor structure resulted from the practice of using problem blocks and from pairs of similar questions. Applying MIRT to a reduced set of FCI items removing blocked items and repeated items produced a 6-factor solution; however, the factors had little relation the general structure of Newtonian mechanics. A theoretical model of the FCI was constructed from expert solutions and fit to the FCI by constraining the MIRT parameter matrix to the theoretical model. Variations on the theoretical model were then explored to identify an optimal model. The optimal model supported the differentiation of Newton's 1st and 2nd law; of one-dimensional and three-dimensional kinematics; and of the principle of the addition of forces from Newton's 2nd law. The model suggested by the authors of the FCI was also fit; the optimal MIRT model was statistically superior

Using Machine Learning to Identify the Most At-Risk Students in Physics Classes

Machine learning algorithms have recently been used to predict students' performance in an introductory physics class. The prediction model classified students as those likely to receive an A or B or students likely to receive a grade of C, D, F or withdraw from the class. Early prediction could better allow the direction of educational interventions and the allocation of educational resources. However, the performance metrics used in that study become unreliable when used to classify whether a student would receive an A, B or C (the ABC outcome) or if they would receive a D, F or withdraw (W) from the class (the DFW outcome) because the outcome is substantially unbalanced with between 10\% to 20\% of the students receiving a D, F, or W. This work presents techniques to adjust the prediction models and alternate model performance metrics more appropriate for unbalanced outcome variables. These techniques were applied to three samples drawn from introductory mechanics classes at two institutions ($N=7184$, $1683$, and $926$). Applying the same methods as the earlier study produced a classifier that was very inaccurate, classifying only 16\% of the DFW cases correctly; tuning the model increased the DFW classification accuracy to 43\%. Using a combination of institutional and in-class data improved DFW accuracy to 53\% by the second week of class. As in the prior study, demographic variables such as gender, underrepresented minority status, first-generation college student status, and low socioeconomic status were not important variables in the final prediction models.Comment: arXiv admin note: substantial text overlap with arXiv:2002.0196